Constraint-based Visual Hand Gesture Analysis

The rapid progress in Virtual Reality technology has demanded more natural and effective human-computer interaction (HCI) for tasks such as display control, navigation and virtual object manipulation. Conventional interface devices such as keyboards and mice are cumbersome to use in Virtual Environments (VE). Bare hand gesturing (especially in combination with speech) is a leading candidate for achieving natural and effective HCI in VE. Visual tracking and recognition of hand/finger gestures are difficult, mainly because of the large degrees of freedom (DoF) of the hand/finger configuration (joint angles). However, the hand/finger configuration and movement are highly constrained. Our research is to find good representations of these constraints, and use them to help vision-based articulated motion capturing, analysis, tracking and recognition (to make them computationally feasible for real-time applications). A crucial part of our research is to characterize hand/finger motion constraints. Based on a large data set collected with a CyberGlove, we developed a learning approach to construct the constraint model by characterizing the structure of a lower dimensional hand configuration space. The data are also used to estimate parameters of the dynamic model of natural hand/finger motions in several application domains including the manipulation of virtual objects. The second part of our research is the efficient visual capturing of hand motion. We utilize the hand motion constraint model to build a system for tracking hand articulation. Since the constraint model provides a priori knowledge of hand motion, our probabilistic motion capturing approach would tolerate the inaccuracy of the constraint model and achieve accurate tracking results. Based on our motion capturing method, a more immersive and convenient gesture interface can be built for a wide range of applications. Although the main emphasis of our research is on hand/finger tracking and gesture recognition, the constraint representations we developed is very useful for animation as well. The third part of our research attacks the difficult problem of automatic initialization in visual tracking, i.e. to find the initial palm pose and finger configuration in the first frame of an image sequence. This task is difficult because the search range of the initial states of the target would be quite huge. In summary, our research aims to attack the challenging problem of visual motion capturing of the highly articulated hand by constructing motion constraint models. We are investigating several fundamental research problems and our approaches are general. The results would thus benefit a wide range of other research problems, as well.